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Large-scale numeracy assessments are intended to facilitate the improvement of 
educational outcomes; however, it is not clear exactly how this is to be achieved. To move 
towards the goal of numeracy for all, it is necessary to systematically address issues that are 
known to be difficult, pervasive and persistent. This paper includes an analysis of an 
'addition of fractions' item from the Australian 2008 Year 7 NAPLAN assessment and 
draws insights that may be generalised to improve overall numeracy. 


The Role of Large-Scale Assessments in the Education System 

The importance of numeracy skills in modern society is clear. People who lack 
sufficient numeracy skills to evaluate whether a financial statement or contract is 
reasonable are vulnerable. Large-scale assessments (LSA’s) function as a means of 
evaluating the performance of the education system as a whole against pre-defined criteria 
and therefore play a role in the national development of numeracy as a matter of 
governance. In the Executive Summary of the Australian Government-initiated Review for 
School Funding, Gonski (2011) states: “... no student in Australia should leave school 
without the basic skills and competencies needed to participate in the workforce and lead 
successful and productive lives” (p. xv). LSA’s might be considered as a governmental 
response to the evolving needs of modem society by obtaining data on which to base 
policy decisions while recognising, as Gonski (2011) notes, that these criteria are contained 
within the broader goals of “young Australians becoming successful learners, confident 
and creative individuals, and active and informed citizens” (p. 217). The problem that this 
paper seeks to address is how the data from large-scale assessments can be used to attain 
educational improvement. 

In Australia in 2008, the National Assessment Program-Literacy and Numeracy 
(NAPLAN) assessment was introduced for all school children in Years 3, 5, 7 and 9 
(Australian Curriculum, Assessment and Reporting Authority (ACARA), 2011). The 
aggregated national data are reported annually, providing information on children’s 
achievement against a variety of factors including location, gender, Indigeneity and 
parental education. The NAPLAN assessments are intended, at least in part, to facilitate 
systemic improvement over the long term: 

It is important that there be consistent and well understood measures of student achievement around 
the country, and that the outcomes of these assessments be used to inform future policy 
development, resource allocation, curriculum planning and, where necessary, intervention 
programs. (ACARA, 2011) 

As well as providing feedback about the education system as a whole, NAPLAN is also 
intended to provide diagnostic feedback at the local school level to drive local 
improvements over a shorter time-scale. These results can be used to identify individuals 
who require intervention by noting low scores, or progress that was slower than might be 
expected from previous assessments. The Ministerial Council on Education, Employment, 
Training and Youth Affairs (MCEETYA) (2007) highlighted that NAPLAN’s diagnostic 
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capability was an important part of its purpose by choosing May as the month in which to 
hold the assessment: “It is early enough in the year for the test results to be used as a 
diagnostic tool” (p. 1). To facilitate this, detailed results for each child are reported to the 
school. Because the NAPLAN assessment measures the cumulative effect of children’s 
mathematical education to date, this process is less useful for providing a focus for 
reflection on the individual teacher practices at that grade level. 

There is much useful information in the NAPLAN data; however, using the data 
constructively poses a challenge (Leder, 2012). Scores provide a general picture of 
numerical achievement, while individual items provide an opportunity to gain insight on 
children’s thinking, and therefore reflect on what additional experiences would be helpful 
for children. This paper focuses on the results of a single item to examine what might be 
learned from them. 

Although the following observation by Black and Wiliam (2006) was made in the 
context of children learning in a classroom, it was framed as a general statement that might 
be applied in a variety of contexts: 

When anyone is trying to learn, feedback about the effort has three elements: recognition of the 
desired goal, evidence about present position, and some understanding of a way to close the gap 
between the two. All three must be understood to some degree by anyone before he or she can take 
action to improve learning, (p. 6) 

These three elements are discussed here from the perspective of a teacher wishing to 
learn from LSA results. The first element, the desired goal, is to develop children’s 
numeracy skills to a greater extent. The assessment items themselves convey information 
about at least some aspects of the desired goal, an aspect that will be discussed later in this 
paper. The second element, evidence about the present position of children’s attainment, is 
provided by the LSA results. However, LSA’s themselves provide little information 
regarding the third element, the understanding of a way to close the gap. While this 
understanding may reside in the accumulated knowledge of teachers and education 
researchers, in terms of the formal processes involved in LSA’s, the application of this 
knowledge remains ad hoc, and left to individual schools and teachers. 

An ad hoc approach is unlikely to realise the goal of an education system where all 
children attain a rigorous mathematical education as this goal is yet to be achieved 
anywhere. The international and long-standing difficulty of achieving high levels of 
numeracy for any population suggests that the goal is difficult and systemic. This paper 
makes the assumption that substantial progress will require deep rather than shallow 
teaching. It is likely to be necessary to address issues that have already been recognised as 
difficult by teachers and in the research literature. 

This paper sets out to draw on the existing research literature and provide an 
illustration of how Black and Wiliam’s (2006) three elements of learning can be applied to 
NAPLAN results, using a single NAPLAN item on the topic of fraction addition as an 
illustrative example. The general area is recognised as being difficult and important: 

The most important foundational skill not presently developed appears to be proficiency with 
fractions (including decimals, percent, and negative fractions). The teaching of fractions must be 
acknowledged as critically important and improved before an increase in student achievement in 
algebra can be expected. (National Mathematics Advisory Panel, 2008, p. 18) 

This section has described NAPLAN as providing the evidence of the present position 
of children’s numeracy attainment, as part of a larger process of systemically improving 
the outcomes of mathematics education. This paper argues that NAPLAN items draw 
attention to issues of such a long-standing nature that they have become part of the 
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background scenery. The NAPLAN data provide an opportunity to consider children’s 
attainment at an aggregate level, and to draw on existing research to provide understanding 
on how to improve outcomes. The argument is illustrated by use of an item that assessed 
addition of fractions, which is a topic known to be difficult. The interpretation of the 
results of such an item relies on knowledge of children’s understanding of fractions. The 
following section outlines the existing literature relating to this topic. 

Literature Review 

The history of research on children's errors involving items dealing with fractions is 
interesting. Brueckner (1928) reported on a large, detailed study of children's errors with 
fractions undertaken with the aim of comprehensively cataloguing the types of errors that 
children make. Hundreds of children from classes from Grades 4 to 8 in six Minnesota 
elementary schools were given a wide variety of fraction items to complete and their errors 
were recorded and classified. The study was extensive; on the topic of addition of fractions 
alone, 25,000 examples of written work were collected and analysed. Brueckner’s study 
was concerned with children's ability to apply processes rather than children's 
understanding. Despite this, Brueckner’s observation that children were not presented with 
a wide enough variety of drill materials is echoed in modem calls for children to 
experience a wider variety of tasks. 

It is interesting to note that the error in which the numerators were added together and 
the denominators were added together was classified as a “Lack of comprehension of 
process involved” (Brueckner, 1928, p. 762), whereas more recent researchers might be 
more likely to identify this error as a misapplication of additive processes (e.g., Brown & 
Quinn, 2006). This type of error may arise from a mis-generalisation of an understanding 
of fractions as parts of a set of objects. Smith (2002) pointed out that older children who 
continue to make this error justify their reasoning along the following lines: 3 represents 
three parts out of five, and 2 represents two parts out of five. Adding the fractions together 
is then construed as five parts out of a possible eight parts. 

This error suggests inappropriate generalisation from experiences with combining 
collections of objects. Learning algorithms prior to having understood underlying 
principles is known to be counter-productive (Clarke, 2005). Hart (1980) remarked that we 
“have ample proof that they [algorithms] are not remembered or sometimes remembered in 
a form that was never taught, e.g. to add two fractions, add the tops and add the bottoms” 
(p. 212). While this faulty algorithm may not be taught explicitly, it is quite possible that 
children are mis-generalising from marked tests that they see frequently in the classroom. 
For many mathematics tests and assignments, the number of marks attained for each 
question is represented as the numerator of a fraction, with the denominator being the total 
possible marks for that question. The score for an entire test is found by adding all of the 
numerators, to give the score attained. This is often expressed as a fraction of the possible 
score, which is obtained by from the sum of the maximum possible marks for each 
individual item, previously expressed as the denominators for the individual item scores. In 
other words, although teachers are not explicitly teaching the faulty algorithm, it is not a 
surprise that children construct it for themselves. 

The fractional representation of the mark for each question is not, in itself, problematic. 
The problem lies in the apparent addition of fractions that do not represent the same 
quantity. It would seem that many children lack a sound understanding of the nature of a 
fractional number. At the most basic level, fractions represent a quantity that can be 
represented on a number line (Kilpatrick, Swafford, & Findell, 2001; Siegler, Thompson, 
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& Schneider, 2011). However, many students, and some teachers, lack a well-grounded 
sense of the magnitude of fractions (Clarke & Roche, 2009; Siegler, Fazio, Bailey, & 
Zhou, 2013). 


Method 

This paper sets out to illustrate that NAPLAN assessments have the capability of 
focusing attention on difficulties children experience with numeracy and thus lead to 
constructive implications for teachers. The secondary analysis presented in this paper 
presents an in-depth analysis of anonymised children’s responses to a single item, Item 25, 
from the 2008 Year 7 NAPLAN Numeracy non-calculator assessment. This item was 
selected because it possessed the following features: 

• The topic, addition of fractions, has been observed in the literature to be 
difficult. 

• A higher proportion of children than had chosen a particular incorrect response 
than had chosen the correct response. 

• The popular incorrect answer was implausible by applying a common-sense 
measure of magnitude. 

This unusual combination of features is interesting, and is suggestive that an analysis may 
provide insights into difficulties children may have either with this topic, or perhaps more 
general difficulties. 

The interpretation of the inherently limited information available from the NAPLAN 
data was informed by drawing on the research literature on this topic. NAPLAN data are 
used and reproduced with permission of the Victorian Curriculum and Assessment 
Authority (VCAA). Analysis and findings using that data are not connected with or 
endorsed by the VCAA. 

Analysis of Children’s Responses to Fractions Item 

This section presents an analysis of Year 7 children’s responses to Item 25 on the 
Victorian NAPLAN Numeracy non-calculator assessment. Item 25 was a multiple choice 
item that was answered correctly by 35% of children: 

2 

A garden centre sells a potting mix made up of soil, compost and sand. Soil makes up r of the mix 
1 

and compost makes up ^ of the mix. What fraction of the potting mix is sand? 

The options were: (correct), — and y The option y, which corresponds to 

adding the numerators and the denominators to arrive at an answer, was chosen by 37% of 
children. The remaining two options were chosen by 12% and 13% of children 
respectively. This item is interesting because of the high proportion of children who added 
the numerators and the denominators of the two fractions without regard to the question, 
suggesting that some children are generalising from their instrumental understanding of 
adding whole numbers to apply the same process to adding fractions (Brown & Quinn, 
2006). 

The potting mix item involved both problem-solving skills and multiplicative thinking 
in the use of fractions. The apparent difficulty of this item is dependent on whether it is 
approached as an exercise with fraction operations or as a problem-solving exercise. 
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Multiple choice items ask questions of the form: Given the answer is one of the options 
shown, which one of these options answers the given question ? (Morley, 2013). This 
section discusses a possible problem-solving approach in which the question might be 

1 3 5 4 2 

restated as: Which of the following options: — , - , — and -could be added with -and 

12 7 12 7 3 


— to make 1? 

4 

There are two aspects of fraction understanding required to answer this question using 
a problem-solving approach. The first is that the sum of the fractions given, including the 
unknown fraction of sand, is equal to one. The second is a sense of the size of the fractions. 

2 1 

Since the amount of soil in the mix is - , the remainder of the mix totals - . The addition 

3 3 

of compost allows the inference that the only plausible option is — , since this is the only 


option that is less than 


1 

3 


. It is only necessary to consider the first term, and not at all 


necessary to perform the calculation - + - . From a diagnostic perspective, it would appear 


that most children who answered this item incorrectly children are not approaching this 
item by considering what a plausible answer might be in terms of magnitude. Either 
children have not yet acquired a sense of the size of such fractions, as suggested by 
previous research (e.g., Clarke & Roche, 2009), or they are not approaching the item by 
considering what a plausible answer might be prior to performing any calculations. 

An alternative approach to the item that may have been taken, as an exercise in 
operating with fractions, is discussed here. In this case, obtaining the correct answer 


requires finding the sum of 


- and 

3 


— which is 
4 


n 

12 


Since the required amount, when added 


to would equal one, subtracting 


from one yields the correct answer, 


12 


. Finding 


2 1 

the sum of — and — might present a high cognitive load for children who are under time 

pressure unless they are fluent with fractions and familiar with this type of problem to the 
point where it is routine. 

3 

The fact that 37% of children chose the option — , corresponding to adding the 


numerators and the denominators, suggests that children did not consider the magnitudes 
of each of the fractions stated in the item question as numbers with individual magnitudes. 
This finding supports Siegler et al.’s (2013) contention that: 


Learning to accurately represent and arithmetically combine the magnitudes of all types of real 
numbers - whole numbers and fractions; positives and negatives; common fractions, decimals, and 
percentages - is thus inherently central to numerical development 


This analysis has provided findings from NAPLAN data that support the need for 
robust teaching practices regarding the numbers, particularly the introduction of fractional 
numbers. The next section discusses the implications for teachers of the above analysis and 
then discusses the limitations of individual teachers to effect the desired educational 
changes. 


481 



Morley 


Implications of Analysis for Teachers 

This section discusses the implications for teachers that arise from the analysis of the 
addition of fractions item. Haladyna, Downing and Rodriguez (2002) reviewed multiple 
choice item-writing guidelines and found that among those universally endorsed by 
textbooks were that distractors should be plausible and reflect common errors by students. 
The item analysed earlier clearly reflects common errors and therefore may reasonably be 
considered plausible to the children involved. Yet, this distractor would not be considered 
plausible if the magnitude of the fractions were considered. Whether this distractor could 
be considered plausible or not is therefore dependent on the children’s expertise with the 
magnitude of fractions. 

The data are consistent with Siegler et al.’s (2013) position that “interventions that 
improve fraction magnitude representations also improve other mathematical capabilities” 
(p. 17). Clarke (2005) suggests that the early introduction of written algorithms that 
operate on a number from the digit representing the smallest magnitude to the digit 
representing the largest magnitude interferes with children’s natural thinking of magnitude: 

They tend to lead to blind acceptance of results and over-zealous applications. Given the focus on 

procedures that require little thinking, children often use an algorithm when it is not at all necessary. 

(P- 94 ). 

Blind acceptance of results is antithetical to the goal of numeracy. Numeracy, at least 
in part, involves being able to detect whether a given result is plausible or not. The 
desirability of this skill is not limited by age or proficiency with mathematics. Even for 
people who are proficient with complex calculation, such as engineers, it is important to 
evaluate the plausibility of a result, whether the calculation is done by hand or by 
computer. It would appear that a consideration of what results might be accepted as 
plausible prior to performing calculations is a neglected area in Victorian mathematics 
education and therefore addressing this issue offers scope for the general improvement of 
numeracy outcomes. 

Implications for the National Development of Numeracy in Australia. 

This section discusses of the implications of this analysis on the broader question of the 
use of NAPLAN data to effect system wide improvements. Large-scale assessments such 
as NAPLAN are intended to provide direct feedback of children’s mathematical 
achievement to teachers and thereby drive improvements to teacher practices. While 
assessments may exemplify aspects of the curriculum to an extent, generating 
improvements through feedback to teachers of the large-scale assessment results remains 
ad-hoc in approach, rather than systematic, because beneficial changes that might be made 
as a result of individual teachers or schools are not systematically identified and 
propagated back through the system. 

Lor example, assume, for the sake of argument, the hypothetical ideal situation: 

• feedback to teachers was clear and instantaneous; 

• the current teacher was the only teacher whose practices were involved; 

• the action required to attain desired outcomes was clear; 

• a particular teacher managed to incorporate the feedback perfectly; and 

• the subsequent assessment results reflected the improved teaching. 

Even with these ideal conditions, the improvements that would result would remain local 
to the school unless successful classroom innovations were identified so that they might 
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propagate through the school system, as a teachers’ classroom practice does not extend far 
beyond their own class. While this occurs to some extent through the professional 
literature, conferences and other professional activities, it is not incorporated directly 
through the NAPLAN system. Further, competition between schools fostered by the 
posting of schools’ NAPLAN attainment on the My School website may inhibit sharing. 
Large-scale assessments provide not only the opportunity to identify schools requiring 
additional support, but there is also the potential opportunity to identify and share practices 
that are found to be exemplary, as yet largely untapped. The required changes are system- 
wide, not local, and the feedback from assessments have implications for teaching 
practices beyond the current teachers of the assessed cohort. 

Conclusion 

The goal of improving educational outcomes for all is a daunting one. This paper 
addressed how the data from large-scale assessments can be used to attain educational 
improvement, illustrated by an analysis of an item that assessed proficiency with fractions. 

Drawing on the research literature, the analysis of a fractions item from a Year 7 
Numeracy assessment suggested that interventions that strengthen children’s understanding 
of the magnitudes of fractional numbers may be beneficial (e.g. Siegler et al., 2013). This 
finding suggests that difficulties that may be evident in Year 7 are an issue in the short 
term for the current teacher of the child, but for sustained systemic improvement they are 
also an issue for teachers in earlier years. The implication for the education community is 
that findings from NAPLAN results have relevance for a wider section of the education 
community than the current teacher. 

Generalising the findings from the analysis of the fractional item it is possible that 
interventions that encourage children to consider what answers may or may not be 
plausible for any given question may help develop children’s ability to connect the results 
that they obtain from calculations with their broader understanding of numerical concepts. 
Further research in this area may be beneficial. 

Children’s difficulty with addition of fractions has been noted in the literature for many 
decades. The persistence of children’s difficulty in adding fractions is one example that 
provides evidence that the attainment of such understanding is non-trivial. Drawing on 
Black and Wiliam's (2006) three elements required for learning, large-scale assessments 
such as NAPLAN may clarify the first element, specific attainment goals, and directly 
address the second element of providing evidence of children’s current position. However, 
LSA’s do little to address the third element: attaining an understanding of how to close the 
gap between the goal and the current position. This paper, therefore, recommends further 
research on the processes involved with large-scale assessments to encourage a more 
systemic approach to improving education rather than relying on ad hoc efforts duplicated 
throughout the nation. 
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